Example: Hyperparameter tuning¶

---

This example shows an advanced example on how to optimize your model's hyperparameters for multi-metric runs.

Import the breast cancer dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict whether a patient has breast cancer or not.

Load the data¶

In [1]:

# Import packages
from sklearn.datasets import load_breast_cancer
from atom import ATOMClassifier

# Import packages from sklearn.datasets import load_breast_cancer from atom import ATOMClassifier

In [2]:

# Load the data
X, y = load_breast_cancer(return_X_y=True)

# Load the data X, y = load_breast_cancer(return_X_y=True)

Run the pipeline¶

In [3]:

# Initialize atom
atom = ATOMClassifier(X, y, verbose=2, random_state=1)

# Initialize atom atom = ATOMClassifier(X, y, verbose=2, random_state=1)

<< ================== ATOM ================== >>
Algorithm task: binary classification.

Dataset stats ==================== >>
Shape: (569, 31)
Memory: 138.96 kB
Scaled: False
Outlier values: 169 (1.2%)
-------------------------------------
Train set size: 456
Test set size: 113
-------------------------------------
|   |     dataset |       train |        test |
| - | ----------- | ----------- | ----------- |
| 0 |   212 (1.0) |   170 (1.0) |    42 (1.0) |
| 1 |   357 (1.7) |   286 (1.7) |    71 (1.7) |

In [4]:

# Train a LogisticRegression model on two metrics
atom.run(models="LR", metric=["f1", "ap"], n_trials=10)

# Train a LogisticRegression model on two metrics atom.run(models="LR", metric=["f1", "ap"], n_trials=10)

Training ========================= >>
Models: LR
Metric: f1, average_precision


Running hyperparameter tuning for LogisticRegression...
| trial | penalty |       C |  solver | max_iter | l1_ratio |      f1 | best_f1 | average_precision | best_average_precision | time_trial | time_ht |    state |
| ----- | ------- | ------- | ------- | -------- | -------- | ------- | ------- | ----------------- | ---------------------- | ---------- | ------- | -------- |
| 0     |      l2 |  0.0054 |    saga |      480 |      --- |  0.9744 |  0.9744 |            0.9991 |                 0.9991 |     0.017s |  0.017s | COMPLETE |
| 1     |      l2 |   0.122 |    saga |      380 |      --- |  0.9828 |  0.9828 |            0.9859 |                 0.9991 |     0.024s |  0.041s | COMPLETE |
| 2     |      l2 |  0.0071 |     sag |      720 |      --- |  0.9744 |  0.9828 |            0.9947 |                 0.9991 |     0.014s |  0.055s | COMPLETE |
| 3     |      l2 | 87.9641 | libli.. |      920 |      --- |  0.9464 |  0.9828 |            0.9591 |                 0.9991 |     0.013s |  0.068s | COMPLETE |
| 4     |    none |     --- |     sag |      630 |      --- |  0.9744 |  0.9828 |            0.9631 |                 0.9991 |     0.071s |  0.139s | COMPLETE |
| 5     |      l2 |  0.0018 |     sag |      920 |      --- |  0.9256 |  0.9828 |            0.9921 |                 0.9991 |     0.015s |  0.154s | COMPLETE |
| 6     |      l2 | 43.4053 |     sag |      780 |      --- |  0.9483 |  0.9828 |            0.9686 |                 0.9991 |     0.085s |  0.239s | COMPLETE |
| 7     |      l2 |  2.0759 | libli.. |      470 |      --- |  0.9655 |  0.9828 |             0.992 |                 0.9991 |     0.013s |  0.252s | COMPLETE |
| 8     |      l2 |   0.043 |     sag |      110 |      --- |  0.9744 |  0.9828 |            0.9971 |                 0.9991 |     0.014s |  0.267s | COMPLETE |
| 9     |      l2 | 46.0233 |    saga |      740 |      --- |  0.9825 |  0.9828 |             0.988 |                 0.9991 |     0.086s |  0.353s | COMPLETE |
Hyperparameter tuning ---------------------------
Best trial --> 0
Best parameters:
 --> penalty: l2
 --> C: 0.0054
 --> solver: saga
 --> max_iter: 480
Best evaluation --> f1: 0.9744   average_precision: 0.9991
Time elapsed: 0.353s
Fit ---------------------------------------------
Train evaluation --> f1: 0.963   average_precision: 0.9941
Test evaluation --> f1: 0.966   average_precision: 0.9962
Time elapsed: 0.026s
-------------------------------------------------
Total time: 0.379s


Final results ==================== >>
Total time: 0.397s
-------------------------------------
LogisticRegression --> f1: 0.966   average_precision: 0.9962

In [5]:

# For multi-metric runs, the selected best trial is the first in the Pareto front
atom.lr.best_trial

# For multi-metric runs, the selected best trial is the first in the Pareto front atom.lr.best_trial

Out[5]:

FrozenTrial(number=0, values=[0.9743589743589743, 0.9990764494418141], datetime_start=datetime.datetime(2022, 9, 22, 10, 37, 10, 64858), datetime_complete=datetime.datetime(2022, 9, 22, 10, 37, 10, 81874), params={'penalty': 'l2', 'C': 0.005417257182767969, 'solver': 'saga', 'max_iter': 480, 'l1_ratio': 0.7000000000000001}, distributions={'penalty': CategoricalDistribution(choices=('l1', 'l2', 'elasticnet', 'none')), 'C': FloatDistribution(high=100.0, log=True, low=0.001, step=None), 'solver': CategoricalDistribution(choices=('lbfgs', 'newton-cg', 'liblinear', 'sag', 'saga')), 'max_iter': IntDistribution(high=1000, log=False, low=100, step=10), 'l1_ratio': FloatDistribution(high=1.0, log=False, low=0.0, step=0.1)}, user_attrs={'params': {'penalty': 'l2', 'C': 0.0054, 'solver': 'saga', 'max_iter': 480}, 'estimator': LogisticRegression(C=0.0054, max_iter=480, n_jobs=1, random_state=1,
                   solver='saga')}, system_attrs={'nsga2:generation': 0}, intermediate_values={}, trial_id=0, state=TrialState.COMPLETE, value=None)

In [6]:

# Use the plot_pareto_front to select a better option
from optuna.visualization import plot_pareto_front

plot_pareto_front(atom.lr.study)

# Use the plot_pareto_front to select a better option from optuna.visualization import plot_pareto_front plot_pareto_front(atom.lr.study)

In [7]:

# If you are unhappy with the results, it's possible to conitnue the study
atom.lr.hyperparameter_tuning(n_trials=5)

# If you are unhappy with the results, it's possible to conitnue the study atom.lr.hyperparameter_tuning(n_trials=5)

Running hyperparameter tuning for LogisticRegression...
| trial | penalty |       C |  solver | max_iter | l1_ratio |      f1 | best_f1 | average_precision | best_average_precision | time_trial | time_ht |    state |
| ----- | ------- | ------- | ------- | -------- | -------- | ------- | ------- | ----------------- | ---------------------- | ---------- | ------- | -------- |
| 10    |      l2 |  19.959 | libli.. |      630 |      --- |  0.9655 |  0.9828 |            0.9737 |                 0.9991 |     0.021s |  0.373s | COMPLETE |
| 11    |      l2 | 10.8524 | newto.. |      150 |      --- |  0.9735 |  0.9828 |             0.996 |                 0.9991 |     0.018s |  0.392s | COMPLETE |
| 12    |      l2 |  0.6289 | libli.. |      320 |      --- |  0.9821 |  0.9828 |               1.0 |                    1.0 |     0.016s |  0.407s | COMPLETE |
| 13    |      l2 |  0.0158 | libli.. |      160 |      --- |  0.9913 |  0.9913 |            0.9994 |                    1.0 |     0.016s |  0.423s | COMPLETE |
| 14    |      l2 |   0.792 | libli.. |      160 |      --- |  0.9744 |  0.9913 |            0.9956 |                    1.0 |     0.016s |  0.438s | COMPLETE |
Hyperparameter tuning ---------------------------
Best trial --> 12
Best parameters:
 --> penalty: l2
 --> C: 0.6289
 --> solver: liblinear
 --> max_iter: 320
Best evaluation --> f1: 0.9821   average_precision: 1.0
Time elapsed: 0.438s

In [8]:

# The trials attribute gives an overview of the trial results
atom.lr.trials

# The trials attribute gives an overview of the trial results atom.lr.trials

Out[8]:

	params	estimator	score	time_trial	time_ht
trial
0	{'penalty': 'l2', 'C': 0.0054, 'solver': 'saga...	LogisticRegression(C=0.0054, max_iter=480, n_j...	[0.9743589743589743, 0.9990764494418141]	0.017016	0.017016
1	{'penalty': 'l2', 'C': 0.122, 'solver': 'saga'...	LogisticRegression(C=0.122, max_iter=380, n_jo...	[0.9827586206896551, 0.9858721483975268]	0.024022	0.041038
2	{'penalty': 'l2', 'C': 0.0071, 'solver': 'sag'...	LogisticRegression(C=0.0071, max_iter=720, n_j...	[0.9743589743589743, 0.9947192281078494]	0.014012	0.05505
3	{'penalty': 'l2', 'C': 87.9641, 'solver': 'lib...	LogisticRegression(C=87.9641, max_iter=920, n_...	[0.9464285714285715, 0.9590727963935599]	0.013179	0.068229
4	{'penalty': 'none', 'solver': 'sag', 'max_iter...	LogisticRegression(max_iter=630, n_jobs=1, pen...	[0.9743589743589743, 0.9631035970079513]	0.071064	0.139293
5	{'penalty': 'l2', 'C': 0.0018, 'solver': 'sag'...	LogisticRegression(C=0.0018, max_iter=920, n_j...	[0.9256198347107438, 0.9921466959495802]	0.015013	0.154306
6	{'penalty': 'l2', 'C': 43.4053, 'solver': 'sag...	LogisticRegression(C=43.4053, max_iter=780, n_...	[0.9482758620689654, 0.9685587859139686]	0.085077	0.239383
7	{'penalty': 'l2', 'C': 2.0759, 'solver': 'libl...	LogisticRegression(C=2.0759, max_iter=470, n_j...	[0.9655172413793103, 0.9920077193219741]	0.013012	0.252395
8	{'penalty': 'l2', 'C': 0.043, 'solver': 'sag',...	LogisticRegression(C=0.043, max_iter=110, n_jo...	[0.9743589743589743, 0.9970693113085392]	0.014181	0.266576
9	{'penalty': 'l2', 'C': 46.0233, 'solver': 'sag...	LogisticRegression(C=46.0233, max_iter=740, n_...	[0.9824561403508771, 0.9879894597165679]	0.086391	0.352967
10	{'penalty': 'l2', 'C': 19.959, 'solver': 'libl...	LogisticRegression(C=19.959, max_iter=630, n_j...	[0.9655172413793103, 0.9736590712255391]	0.020528	0.373495
11	{'penalty': 'l2', 'C': 10.8524, 'solver': 'new...	LogisticRegression(C=10.8524, max_iter=150, n_...	[0.9734513274336283, 0.9959569642562589]	0.018023	0.391518
12	{'penalty': 'l2', 'C': 0.6289, 'solver': 'libl...	LogisticRegression(C=0.6289, max_iter=320, n_j...	[0.9821428571428572, 1.0000000000000002]	0.015626	0.407144
13	{'penalty': 'l2', 'C': 0.0158, 'solver': 'libl...	LogisticRegression(C=0.0158, max_iter=160, n_j...	[0.9913043478260869, 0.999389732649834]	0.01563	0.422774
14	{'penalty': 'l2', 'C': 0.792, 'solver': 'libli...	LogisticRegression(C=0.792, max_iter=160, n_jo...	[0.9743589743589743, 0.9955890815002342]	0.015627	0.438401

In [9]:

# Select a custom best trial...
atom.lr.best_trial = 2

# ...and check that the best parameters are now those in the selected trial
atom.lr.best_params

# Select a custom best trial... atom.lr.best_trial = 2 # ...and check that the best parameters are now those in the selected trial atom.lr.best_params

Out[9]:

{'penalty': 'l2', 'C': 0.0071, 'solver': 'sag', 'max_iter': 720}

In [10]:

# Lastly, fit the model on the complete training set 
# using the new combination of hyperparameters
atom.lr.fit()

# Lastly, fit the model on the complete training set # using the new combination of hyperparameters atom.lr.fit()

Fit ---------------------------------------------
Train evaluation --> f1: 0.963   average_precision: 0.9941
Test evaluation --> f1: 0.966   average_precision: 0.9962
Time elapsed: 0.058s

Analyze the results¶

In [11]:

atom.plot_trials()

atom.plot_trials()